Add ibis/xorq analysis backend - #518
Closed
paddymul wants to merge 6 commits into
Closed
Conversation
- Fix IbisAnalysisPipeline: remove premature xorq guard (only require ibis), add schema pre-seeding for dtype, filter None from expression results, add histogram_query_fns support - Add ibis_stats_v2.py with IbisTypingStats, IbisBaseSummaryStats, IbisNumericStats, IbisComputedSummaryStats, IbisHistogramStats - Add xorq optional dependency group to pyproject.toml - Add 23 tests using ibis.memtable (no xorq needed for local tests) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pre-push hook required ruff format on the new ibis files; regenerated the lock so xorq picks up 0.3.21 (was 0.3.10). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
paddymul
force-pushed
the
feat/paf-v2-ibis-v2
branch
from
April 30, 2026 12:07
5f52154 to
aa8da8c
Compare
Contributor
📦 TestPyPI package publishedpip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.13.5.dev25165205118or with uv: uv pip install --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo==0.13.5.dev25165205118MCP server for Claude Codeclaude mcp add buckaroo-table -- uvx --from "buckaroo[mcp]==0.13.5.dev25165205118" --index-strategy unsafe-best-match --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ buckaroo-table📖 Docs preview🎨 Storybook preview |
Replaces the existing v1-style class tests with tests targeting the yet-to-exist IbisStatPipeline + @stat-based ibis_stats_v2 module. Adds coverage that was missing in v1: - Live histogram (numeric + categorical) — was dead code - Structured error capture: a bad @stat surfaces as a StatError, not a silent stat dropout - DAG validation at pipeline construction - Constant-column histogram edge case Will fail collection until ibis_stat_pipeline.py exists. Implementation in the next commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the v1-style IbisAnalysis class hierarchy with @stat-decorated functions executed through a new IbisStatPipeline that mirrors the typed-DAG semantics of StatPipeline. Two-phase execution: 1. Batch aggregate — every @stat with an IbisColumn parameter is folded into a single table.aggregate() query and executed once. 2. Per-column post-batch — computed stats and IbisTable-param stats (histograms) run through the standard typed-DAG executor. Resolves the issues called out in the adversarial review: - Histogram is live (was dead code in v1: IbisHistogramStats was defined but never added to IBIS_ANALYSIS). Numeric and categorical paths now have test coverage that asserts populated histograms. - Structured error capture replaces the three `except Exception: continue` blocks. Failures inside an @stat or in the batch query surface as StatError entries; downstream computed stats still see UpstreamError propagation. Tests assert that an intentionally bad @stat shows up in the returned errors list. - DAG ordering is implicit. requires_summary is no longer a comment — parameter names ARE the dependency declaration, validated at construction via build_typed_dag. - Naming matches the v2 pattern: filename stays ibis_stats_v2 but the contents now match the @stat / TypedDict style of pd_stats_v2. - Empty-table length=0 fix: the previous `COUNT(c) + SUM(...IS NULL)` expression returned NULL on an empty table because SUM-over-empty is NULL in SQL. Coalesced to 0. Adds IbisColumn / IbisTable marker types to stat_func.RAW_MARKER_TYPES and a small IbisTable injection branch in _execute_stat_func so the v2 pipeline can pass the table through to histogram-style stats. Deletes the v1 IbisAnalysisPipeline / IbisAnalysis (no remaining consumers in buckaroo/ or tests/). 25/25 ibis tests pass; full unit suite passes (modulo the pre-existing unrelated MCP uvx flake on main). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Asserts the DfStats-shaped wrapper that lets DataFlow / BuckarooWidget consume an ibis.Table: .sdf, .errs (v1 ErrDict shape), the verify_analysis_objects classmethod, and .ap.ordered_a_objs. Will fail collection until IbisDfStatsV2 is added in df_stats_v2.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makes the ibis pipeline reachable from the rest of buckaroo:
- Adds IbisDfStatsV2 in df_stats_v2.py mirroring DfStatsV2 /
PlDfStatsV2 (.sdf, .errs, .ap.ordered_a_objs,
verify_analysis_objects). Any DataFlow consumer that takes an
arbitrary DFStatsClass now also accepts the ibis path.
- Adds process_table_v1_compat to IbisStatPipeline so the wrapper
returns the same {(col, stat): (Exception, kls)} ErrDict shape
that AnalysisPipeline produced.
- Adds ordered_a_objs property to IbisStatPipeline so the
add_analysis path matches StatPipeline.
ibis is still imported lazily inside IbisDfStatsV2 — the rest of the
df_stats_v2 module stays ibis-free.
End-to-end test asserts the round-trip (table -> .sdf with length=5
and mean=3.0) and that v1 errs shape matches the existing convention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Closing — GitHub UI was showing a phantom 499-file diff after the rebase, even though the actual diff is 10 files (verified via API and |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IbisAnalysisPipelineto only require ibis (not xorq), adds schema pre-seeding, filters None expressions, adds histogram query supportibis_stats_v2.pywith 5 IbisAnalysis subclasses mirroring the pandas/polars stat classes (typing, base summary, numeric, computed, histogram)xorqoptional dependency group to pyproject.tomlibis.memtable()— no xorq needed for local testingReplaces #508 — clean cherry-pick onto current main (after #515, #516, and #517 dependencies merged).
Test plan
ruff checkpassestest_ibis_stats_v2.py— 23 tests using ibis.memtable()🤖 Generated with Claude Code